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(g) 



The invention relates to an image conversion unit for converting a jQrst image 



with a first resolution into a second image with a second resolution, the image conversion 
unit comprising: 

- a coefficient-calculating means for calculating a first filter coefGcient on 
5 basis of pixel values of the first image; 

- an adaptive filtering means for calculating a second pixel value of the second 
image on basis of a first one of the pixel values of the first image and the first filter 
coefficient. 

The invention further relates to a method of converting a first image with a 
10 first resolution into a second image with a second resolution, the method comprising: 

- calculating a first filter coefficient on basis of pixel values of the first image; 

- calculating a second pixel value of the second image on basis of a fibrst one of 
the pixel values of the first image and the first filter coefficient. 

The invention finilier relates to an image processing apparatus comprising: 
15 - receiving means for receiving a signal corresponding to the Grst image; and 

- the above mentioned image conversion unit for converting a first image into 
a second image. 

20 The advait of HDTV emphasizes the need for spatial up-conversion 

techniques that enable standard definition (SD) video material to be viewed on high 
definition (HD) television (TV) displays. Conventional techniques are linear interpolation 
methods such as bi-linear interpolation and methods using poly-phase low-pass interpolation 
filters. The former is not poptdar in television appUcations because of its low quality, but the 

25 latter is available in commercially available ICs. With the linear methods, the number of 
pixels in tiie frame is increased, but the high frequency part of the spectrum is not extended, 
i.e. the perceived sharpness of the image is not increased. In other words, the capability of the 
display is not fiilly eiqploited. 
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Additional to the conventional linear techniques, a nwnber of non-linear 
algorithms have been proposed to achieve this xip-conversion. Sometimes these techniques 
are referred to as content-based or edge dependent spatial up-conversion. Some of the 
techniques are already available on the consumer electronics market 



•^-embodiment-0f>&6-imag6^onversien^i3ni 

opening paragraph is known from the article **New Edge-Directed Interpolation'% by Xin Li 
et al., in IEEE Transactions on Image Processing, Vol. 10, No 10, October 2001, pp. 1521- 
1S27. In this image conversion unit, the filter coefficirats of an inteipolation up-conversion 
filter are adapted to the local image content. The int^olation up-conversion filter aperture 
uses a fourth order interpolation algorithm as specified in Equation 1: 

F^i2{i + l\20 + l))=j^i^w^^^^ (1) 

with Fffp (iyj) the luminance values of the HD output pixels, F^d {Uj) the luminance values 

of the input pixels and the filter coefficients. The filter coefficients are obtained fix^m a 

largOT aperture using a Least Mean Squares (IMS) optimization procedure. In the cited 
article is explained how the filter coefiSlcients are calculated. The method according to the 
prior art is also explained in connection with Fig. 1 A and Fig. IB, The method aims at 
interpolating along edges rather than across them to prevent blurring. The authors make the 
sensible assumption that edge orientation does not change with scaling. Therefore, the 
coefficients can be approximated firom the SD input image within a local window by using 
the IMS method. 

Although the **New Edge-Directed Interpolation** method according to the 
cited prior art works relatively well in many image parts, there is a problem with the up- 
conversion in areas of the image with abundant detail. 



It is an object of the invention to provide an image conversion unit of the kind 
described in the opening paragraph with an improved performance in areas of the image with 
abundant detail. 

This object of the invention is achieved in that the image conversion unit 
further comprises a further filtering means for filtering the first image resulting in filtered 
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pixel values and the coefificient-calculating meam being a3rratiged to calculate the first filter 
coefficient on basis of the filtered pixel values. Pixel vilues are luminance values or color 
values. An assumption imderlying the prior art algorithm being impl«tnented by means of an 
image conversion unit, Le. that edge orientation does not change with scaling, is no longer 
valid in these areas. This is because, although the coefficients are optimized on a lower 
density grid, there are no means to prevent aliasing at this grid* In the image conversion unit 
according to the invention a further filtering means prior to calculating the filter coe£Bcients 
is included. Notice that this fiurther filtering means is not in the direct path of processing the 
input pixels of the first image into output pixels, i.e. the pixels of the second image, but in the . 
control path to determine the filter coefficieats. Preferably the fiirther filtering means 
comprises a spatial low-pass filter. 

In an embodiment of the image conversion unit according to the invention, the 
spatial low-pass filter has a pass-band substantially corresponding to a quarter of a sampling 
firequency of the first image. This low-pass filter is in correspondence with the Sampling 
Theorem. 

The spatial low-pass filter mi^t be a one-dimensional filter or a cascade of 
two orthogonal filters but altematively a two-dimensional filter is applied. In an embodiment 
of the image conversion unit according to the invention, the spatial low-pass filter is arranged 
to calculate a first one of the filtered pixel values by means of averaging pixel values of a 
block of pixels of the first image. An advantage of this embodiment according to the 
invention is that it is relatively simple. The block of pixels might correspond to e.g. 2*2 or 
2*3or4*4pbcels. 

It is a fiufher object of the invention to provide a method of the kind described 
in the opraong paragraph with an improved performance in areas of the image with abundant 
detail. 

This object of the invention is achieved in that the method fiuther comprises 
filtering the first image resulting in filtered pixel values and calculating flie first filter 
coefficient on basis of the filtered pixel values. 

It is a fiirther object of the invention to provide an image processing apparatus 
of the kind described in the opening paragraph with an improved performance in areas of the 
image with abundant detail. 

This object of the invention is achieved in that the image conversion unit of 
the image processing apparatus fiulher comprises a fiirther filtering means for filtering the 
first image resulting in filtered pixel values and the coefficient-calculating means being 
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arranged to calculate the jBrst filter coefficient on basis of the filtered pixel values. The image 
processing apparatus optionally comprises a display device for displaying the second image. 
The image processing apparatus might e.g. be a TV, a set top box, a VCR (Video Cassette 
Recorder) player or a DVD (Digital Versatile Disk) player. 

5 Modifications of image conversion unit and variations thereof may correspond 

to modifications and variations thereof of the method and of the image processing apparatus 

desenbed; 



10 These and other aspects of the image coiaversion unit, of the method and of the 

image processing apparatus according to the invCTition will become apparent firom and will be 
elucidated with respect to the implementations and embodiments described hereinafter and 
with reference to the accompanying drawings, wherein: 

Fig. 1 A schematically shows an embodiment of the image conversion unit 
1 5 according to the prior art; 

Fig. IB schematically shows a number of pixels to explain the method 
according to the prior art; 

Fig. 2A schematically shows a number of pixels to explain the method 
according to the invention; 
20 Fig. 2B schematically shows an embodiment of the image conversion unit 

according to the invention; 

Fig. 3 A schematically shows as SD input image; 

Fig. 3B schematically shows the SD input image of Fig. 3 A on which pixels 
are added in order to increase the resolution; 
25 Fig. 3C schematically shows the image of Fig. 3B after being rotated over 45 

degrees; 

Fig. 3D schematically shows an HD output image derived firom the SD input 
image of Fig. 3 A; and 

Fig. 4 schematically shows an embodiment of the image processing apparatus 
30 according to the invention. 

Same reference numerals are used to denote similar parts throughout the figures. 



PHNL020828EPP 

5 ^ 10,09.2002 

Fig. 1 A schematically shows an embodiment of the image conversion imit 100 
according to the prior art. The image conversion miit 100 is provided with standard dejBnition 
(SD) images at the input connector 108 and provides high definition (HD) hnages at the 
output coimector 110. The image conversion unit 100 comprises: 

- A pixel acquisition unit 102 which is arranged to acquire a first set of values 
of pixels 1-4 in a first neighborhood of a particular location withia a first one of the SD input 
images which corresponds with the location of an HD output pixel and to acqidre a second 
set of values of pixels 1-16 in a second neighborhood of the particular location within the 
first one of the SD input images; 

- A filter coefficient-calculating unit 106, which is arranged to calculate filter 
coefQcients on basis the first set of values of pixels 1-4 and the second set of values of pixels 
1-16. In other words, the filter coefficients are approximated fix)m tiie SD input image within 
a local window. This is done by using a Least Mean Squares (IMS) method which is 
explained in connection with Fig. IB. 

- An adaptive filtering unit 104 for calculating the value of the HD output 
pixel on basis of the first set of values of pixels 1-4 and the filter coefficients as specified 

. with Equation 1. Hence the filter coefficient-calculating unit 106 is arranged to control the 
adaptive filtering unit 1 04. 

Fig. IB schematically shows a number of pixels 1-16 of an SD input image 
and one HD pixel of an HD output image, to explain the method according to the prior art. 
The HD output pixel is interpolated as a weighted average of 4 pixels 1-4. That means that 
the luminance value of the HD output pixel F^^ results as a weighted sum of its 4 SD 
neigihboring pixels: 

Fm = ^xFsD (1) + ^iPsD (2) + ^^3^^ (3) + w^F^ (4) , (2) 
where F^j, (1) to (4) are the values of the 4 SD input pixels 1-4 and to are the filter 

coefficients to be calculated. The authors of the cited article in which the prior art method is 
described, make the sensible assumption that edge orientation does not change with scaling. 
The consequence of this assumption is that the optimal filter coefficients are the same as 
those to int^olate, on the standard resolution grid: 

- Pixel 1 firam 5, 7, 11, and 4 (that means that pixel 1 can be derived fix)m its 4 

neighbors) 

- Pixel 2 fi-om 6, 8, 3, and 12 

- Pixel 3 firom 9, 2, 13, and 15 



PHNL020828EPP 

m 

^ 6 ^ 10.09.2002 

-Pixel 4 from 1, 10, 14, and 16 
This gives a set of 4 linear equations from which with the LSM-optimization the optimal 4 
filter coefficients to interpolate the HD output pixel are foxmd. 

Denoting M as the pixel set, on the SD -grid, used to calculate the 4 weights, 
5 the Means Square Error (MSB) over set M in the optimization can he written as the sum of 
squared differences between original SD -pixels F^p and interpolated SD -pixels : 



Which in matrix formulation becomes: 

11^ ^ i|2 



USE 



= (4) 

10 Here J contains the SD-pixelsin M (pixel ^^^(u) to F^oM), ^5^(2,1) to F5.^(2,4), 
Fsd(3,1) to Fso(3,4), Fa>(4,l) to Fgoi^A) and C is a 4xM^ matrix whose row is 

composed of the weighted sum of the four diagonal SD -neighbors of each SD -pixels in y . 
The weighted sum of each row describes a pixel F^r , as used in Equation 3. To jBnd the 

minimum MSB , i.e. IMS, the derivation of MSB over w is calculated: 

15 ^ = 0 (5) 

dw 

-2jc+2^*=0 (6) 

w=(c^Cr(c^;) (7) 
By solving Equation 7 the filter coefficients axe found and by using Equation 2 the values of 
the HD output pixels can be calculated, 
20 In this example a window of 4 by 4 pixels is used for the calculation of flie 

filter coefBciCTts. An IMS optimization on a larger window, e.g. 8 by 8 instead of 4 by 4 
gives better residts. 

To frirther elucidate the invention a very simple example implementation of 
the invention is explained, in which the ftirther filtering means is a 4-pixel averaging filter 
25 and the LMS-method is applied on a 5 by 5 block aroimd the HD pixel to be intejpolated. 

Reference is made to Fig. 2A for the numbering of pixels. First, the filtered pixels on the low- 
density grid are calculated, using the average values of their diagonal neighbors, i.e.: 

-Pixel a = (5+6+9+1)74 

- Pixel b = (6+7+1+2)74 
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- Pixel i = (4+12+15+16)74 
Next, the optimal filter coefficients to interpolate pixel HD (at the same position as pixel e) 
5 as a weighted sum of pixels 1, 2, 3, and 4 have to be calculated. To that end, the IMS method 
for solving the optimal interpolation of the 9 filtered pixels, a, b, c,..i &om their 4 diagonal 
neighbors is appUed. For example, pixel e is interpolated firam pixels a, c, g, and i. Fig. 2A 
would have to be extended, as the diagonal ndghbors of pixels a, b, c, d, f, g, h, and I are not 
shown in this Fig. 

IQ Clearly, the J5)erture for the IMS algorithm can be extended to e.g. 5 by 5, or 

7 by 7, and a higher order filter can advantageously replace &e 4-pixel averaging filter. Also 
a symmetrical filter -with an odd number of t^s- can be used with block sizes of 4 by 4, 8 
by 8, etc. Besides that the shape of the aperture does not have to be rectangular. 

Fig. 2B schematically shows an embodiment of the image conversion unit 200 
15 according to the invention. The image conversion unit 200 is provided with standard 

definition (SD) images at the input connector 108 and provides high definition (HD) images 
at the output connector 110. The SD input images have pixel matrices as specified in CCIR- 
601, e.g. 625*720 pixels or 525*720 pixels. The HD output images have pixel matrices with 
e.g. twice or one-and-a-halve times the number of pfacels in horizontal and vertical direction. 
20 The image conversion unit 200 comprises: 

- A pixel acquisition unit 102 which is arranged to acquire a first set of values 
of pixels 1-4 in a first neighborhood of a particular location within a first one of the SD input 
images which corresponds with the location of the output pixel HD and to acquire a second 
set of values of pixels 1-16 in a second neighborhood of the particular; 

25 - A low-pass filter 1 12 for calculating a set of averaged values a, b, c,..i, i.e. 

filtered pixel values a, b, c,..i, on basis of the second set of values of pixels 1-16; 

- A filter coefficient-calculating unit 106 which is arranged to calculate filter 
coefficients on basis of the set of filtered pixels a, b, c,..i. and the second set of values of 
pixels 1-16. In other words, the filter coefficients are approximated firam the filtered SD input 

30 imagewithinalocalwindow.Thisisdonebyusin^aI^MeanSquares(IMS)me1hod 

which is explained in connection with Fig. IB and Fig. 2A; and 

_ An ad^tive filtering unit 104 for calculating a pixel value of an HD output 
image on basis of the second set of values of pixels 1-4. The HD output pixel is calculated as 
the wei^ted sum of the pixels 1-4. 
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The pixel acquisition unit 102, the low-pass filter 1 12, the filter coefficient- 
calculating unit 106 and the adaptive filtering unit 104 may be implemented using one 
processor. NormaUy, these functions are performed under control of a software program 
product. During execution, normally the software program product is loaded into a memory, 
5 like a RAM, and executed from there. The program may be loaded from a background 

memory, like a ROM, hard disk, or magnetically and/or optical storage, or may be loaded via 

a-netvyOTk-Hkgintemet-eptio nally an a ppKcalfoirspgca fic integrated cm;mt provides the 

disclosed ftmctionality. 

To convert an SD input hnage into an HD output image a number of 
10 processing steps are needed. By means of Figs. 3A-3D these processing steps are explained. 
Fig. 3A schematically shows an SD mput image; Fig. 3D schematically shows an HD output 
image derived fi-om the SD input unage of Fig. 3A and Figs. 3B and 3C schematicaUy show 
intermediate results. 

- Fig. 3 A schematically shows an SD input image. Each X-sign correspond 
15 with a respective pixel. 

- Fig. 3B schematically shows the SD iiqjut image of Fig. 3A on which pixels 
are added in order to increase the resolution. The added pixels are indicated with +-signs. 
These added pixels are calculated by means of interpolation of the respective diagonal 
neighbors. The filter coefficients for the interpolation are determined as described in 

20 connection with Fig 2B. 

- Fig. 3C schematically shows the image of Fig. 3B after being rotated over 45 
degrees. The same image conversion unit 200 as being ^plied to calculate the image as 
depicted in Fig. 3B on basis of Fig. 3 A can be used to calculate the image as shown in Fig. 
3D on basis of the image as depicted in Fig. 3B. That means tiiat new pixel values are 

25 calculated by means of interpolation of the respective diagonal neighbors. 

- Fig. 3D schematically shows the final HD output image. The pixels that have 
been added in the last converaion step are indicated with o-signs. 

Fig. 4 schematically shows an embodiment of the image processing apparatus 
400 according to the invention, comprising: 
30 - Receiving means 402 for receiving a signal representmg SD images. The 

signal may be a broadcast signal received via an antenna or cable but may also be a signal 
from a storage device like a VCR (Video Cassette Recorder) or Digital Versatile Disk 
ODVD). The signal is provided at the iiq)ut connector 408; 

- The image conversion unit 404 as described in connection with Fig. 2B; and 
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- A display device 406 for displaying the HD output images of the image 



conversion unit 200. This display device 406 is optional. 

The image processing apparatus 400 might e.g. be a TV. Alternatively the image processing 
apparatus 400 does not comprise the optional display device but provides HD images to an 
5 s^paratus that does comprise a display device 406. Then tihe image processing £^paratus 400 
might be e.g. a set top box, a satellite-tuner, a VCR player or a DVD player. But it migjat also 
be a system being applied by a film-studio or broadcaster. 



limit the invention and that those skilled in the art will be able to design alternative 
1 0 embodiments without departing firom the scope of the appended claims. In the claims, any 

reference signs placed between parentiieses shall not be constructed as limiting the claim. 

The word * comprising' does not exclude the presence of elements or steps not listed in a 

claim. The word "a" or * W preceding an element does not exclude the presence of a 

plurality of such elements. The invention can be implemented by means of hardware 
1 5 comprising several distinct elements and by means of a suitable programmed computer. In 

the unit claims enumerating several means, several of these means can be embodied by one 

and the same item of hardware. 



It should be noted that the above-mentioned embodiments illustrate rather than 
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1 . An image conversion unit for converting a first image with a first resolution 
into a second image with a second resolution, the image conversion unit comprising: 

- a coefficient-calculating means for calcxilating a first filter coefficient on 
basis of pixel values of the first image; 

- an adaptive filtering means for calculating a second pixel value of the second 
image on basis of a first one of the pixel values of the first image and the first filter 
coefficient, characterized in that the image conversion unit further comprises a further 
filtering means for filtering the first image resulting in filtered pixel values and the 
coefficient-<^alculating means being arranged to calculate the first filt^ coefficient on basis of 
the filtered pixel values. 

2. An image conversion xmii as claimed in claim 1 , charact^ized in that the 
further filtering means comprises a spatial low-pass filt^. 

3. An image conversion unit as claimed in claim 2, characterized in that the 
spatial low-pass filter has a pass-band substantially corresponding.to a quarter of a sampling ^ 
firequency of the first image. 

4. An image conversion unit as claimed in claim 2, characterized ia that the 
spatial low-pass filter is arranged to calculate a first one of the filtered pfacel values by means 
of averaging pixel values of a block of pixels of the first image. 

5. A method of converting a first image with a first resolution into a second 
image with a second resolution, the method comprising: 

- calculating a first filter coefficient oh basis of pixel values of the first image; 

- calculating a second pixel value of the second image on basis of a first one of 
the pixel values of the first image and the first filter coefficient, charactdized in that the 
method further comprises filtering the first hnage resulting in filtered pixel values and 
calculating the first filter coefficient on basis of the filtered pixel values. 
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6. An image processing ^paratus comprising: 

- receiving means for receiving a signal corresponding to a first image; and 

- the image conversion unit for converting the first image into a second image, 
as claimed in claim 1. 



An image proc essing apparat 



ifuismf 



display device for displaying the second image. 
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ABSTRACT: It Oft 2002 



An image conversion unit (200) for converting a &st image with a first 
resolution into a second image with a second resolution being higher than the first resolution, 
comprises a filter (1 12) to provide filtered pixel values of the first image to a coefficient- 
calculating means (106) for calculating a first filter coefficient on basis of the filtered pixel 
values. The coefficient-calculating means (106) is arranged to control an adaptive filtering 
means (104) for calculating a second pixel value of the second image on basis of a first pixel 
value of the first image and the first filter coefficient. 



Fig. 2B 
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